Yeah. Okay. Okay. Okay. Yeah. It is the initialisation of. Yeah. But then it is iterative. Okay. So you you should first train the P_L_S_A_ stuff, then you initialise this link uh spreading thing, and then you iterate over okay. And it converge and so you have the boost uh initial P_L_S_A_ and P_L_S_A_ plus the link thing then. Yeah. Mm. Yeah. So th yeah, you you you were saying that you would like to evaluate that now. Yeah, right. Hmm. But but maybe you you could have some some other labelli li or the task like if if you imagine you have some labelling on of of the web pages like like like for example you you look at at at at at a set of pages, and you have the m the directory entries of these pages, so so f like Yahoo or entries, so you have categories and and entries, so for example you have I dunno, cars, S_U_V_, etcetera. So what you would like would be that two pages on in the same um category should be closer than than two pages being spread across different categories. But you you would not use this you will not use the Yahoo directory or n the directory in the training step of i of the model, okay. This is something you have just for evaluation, and then for example you you your task would be to say I have some page and and I know where it is in the in the in the directory, and I have another page, I don't know where it is, and I would like to to know uh whether it is in the same category as the first one, just by comparing their uh aspects. If they have the same uh yeah. So basically what you would like to check if is is is there some similarity in the in the in the aspect distribution, tells me uh something about uh the sim the the fact that they could be on the same category in the the this end end label categories. Yeah. Yeah. Yeah, yeah. Yeah. No no no no no. How would So so so w what what you would have have basically would be to have a directory which give you labels for some part of the documents. Okay, so this is a directory and this is some some some some web page, okay. So the labelled one, okay. And you have some page which are unlabelled but you have links between those two. So what what you would do is that you would try and P_L_S_A_ and and uh and use a model over that. Then then you would like to for example what f what could be very simple would be just to then to match each pages like that. So let's say you have you have a a page which is labelled, and you say okay, the the label page two should get should be the one w which it is the most similar to, or any kind of classification rules you can imagine. And so and so you say okay, what what's if I from from this page which have the similar so so the the criterion would be to compare those two distribution, okay, and then and then you would like to assign to this document too where you don't know the labels the same one as this one. And and here, that you use only for test, so this is your. Then you can evaluate how wrong you are or cu how close you are from the real assignment, the right directory category. In this way I think it's fair if you kn if you know only if you know only that and that and what you would like to discover is this. This is a r like a real task. Like like if if if this directory uh uh company would like to extend this directory to new pages without the hassle of having human labelling. Yeah, you have t you use this one. You you only have the labels for some uh small part of the So So you you would train it you this is this is t t all all those links are known. The links between some labelled document and unlabelled document. I d I think you can do it as a two step process. First you you train uh well maybe s it wou could be better if if if you take it into account. But but maybe you can you can do just a two step process, first one you have this uh unsupervised task over both P_L_ and P_U_, and then you from these unsupervised starts you can know which document of P_L_ is close from the document of P_U_ or the reverse. And you can infer some label to a document of P_L_. Yeah. Yeah yeah. Yeah, but Yeah. Yeah. But but here you can you you say given a random page of P_U_, which of those two model is more likely to to to to to say for example that thi the page of P_L_ which is the most similar, so you so so from this so you take any distribution comparison things, so so Hoffmann in his paper was doing just cosine similarity, but you can do anything. So so you just compare uh those two distributions so of D_ one and D_ two, okay? And so with that you can you can determine for example for a document o of the set two to the most similar So you you you you select D_ one so in this set which is the most similar to D_ two according to the distribution from one or the other model and then and then for this document you automatically assign to it the label of the document uh one and you check with this which is your uh your your if you are right or not. This is what what this is very related to what Frank did with the with the keyword uh image annotation. So you would have a set of keywords, the set of image label with keywords, and he has a set of unlabelled image with some, and it would check whether the the label he assigns, so in this case annotation word h are correct. And I think it's fair uh it's fair uh setup. So maybe this classification rule is not the best. There there's some might be some work to do, but the general idea, I think it's it's fair evaluations, it's a task. That can be dep Yeah. Yeah. No, but according to this directory yeah. yeah. I I I think it's w Yeah. Yeah. Yeah. What's good with the directory is that you have a good coverage of various topic humans humans can label. But then if you want to detect different languages, this is another task. I think the task of this latent model is to discover some some semantic like some topic proximity between documents, and here this la this directory is doing the same thing, like associating some topics with some documents manually. So comparing them is is is is good setup with respect to the goal. I compare those two, like the P_L_S_A_ and the one over this same task of assigning good directory. I don't know if if it if it will provide good result result or not, but it is clearly a benchmark for those two goals. On on the infra yeah, yeah. Yeah. Like bag o yeah. Yeah, you can have tree setups, uh yeah. Yeah, basic cosine over the P_ of T_ given D_, so over all this uh empirical distribution. And the two uh aspect uh models. We have to repeat over and over. This is like advertisement. No no, it's joke. No, it's jus Yeah, yeah. This is really yeah, yeah. Yeah. Yeah. Yeah. Yeah. Yeah. But the point is that the Z_ the the the aspect like the Z_ are are are come unsupervisely. It's it's more like if you were giving the people you you said to them okay, uh divide me this bunch of documents into five topics, and you don't give them the topics a priori, you know. So the two person could come up with very different one is very interesting i in Yeah. Right. Yeah. So I I think I think it's yeah. Yeah. Yeah. But eve ev even someone would would provide very different labels, like like someone would say it's a page in English, someone would say it's a it's a it's a it's a it's a web page uh wh which has very few uh information in it or anything. So I think there's no hard clustering, and what's good with those model is that they they can they they are intended to provide you some some kind of way to compare two documents rather than to give you pre-defined sets of documents. This is not hard clustering. The o I I don't think one one one aspect or one latent topic has any sense by his own. It's when you have the word set and the word distribution that you c you can tell something. If you just take one of them alone, so you take all the P_ of Z_ given D_ and all of the P_ of W_ given giv given Z_, you you you can do anything with that. It's it's the whole set. I would say an an and this is oh it it should be evaluated, meaning that like for example uh the the example about marbles is that the point what you can learn from all these sets made by people is that is that those two those two marble would be consider as more likely to be in the same sets, because they they share some common properties which are not seen as the same for every one. But they share some common properties that people will notice. And if you look at the two the d all possible pairs of marble, you would have some mu much more agreement between people than than if you look at the predefined sets people would would just cut at some precise uh point. You know what's Yeah. Yeah. Well you don't you don't really have two clustering no no no no, you don't you don't really have two clustering. No no no no, you don't have two clustering of of of marbles. What you have is y is you have is you have one clustering of marble, which is this one, and then you have lots of people telling you I consider those marbles as very uh have to have very salient feature that they share in common, I I c I would like them to be in the same cluster. And you c and and and those two, so and one of them is labelled, and the other is not labelled. So you look at that, you say oh, someone told this one was very similar, and this one is said to be blue for example, I dunno if it's the label. So I put it in the blue bag. But but you don't you never use the blue bag, you don't ask the people to look at the blue bag. You just look at the people you you just have the people to compare those two marbles. And then it happens that some of them have been clustered. And and Yeah. Yeah. Yeah, this is I reverse both role. The this is the players. I i no no no no no. It comes up with not with the clustering but with some way to compare two marbles, okay. Some would say would say I consider those two ones are completely dissimilar, and the other ones say they are very similar. And if you find that in your unl unlabelled clustering those two uh balls uh marbles happen to be in the same uh cluster, you said the user algorithm might be wrong. But you average that over lots of documents. Yeah. Yeah. Yeah. Yeah. They make sense only so global optimisation, the likelihood is optimisable globally over the whole aspects, etcetera. So it's it's you never ask the model to do hard clustering of documents, you just uh ask uh it to find a distribution over aspects such that a document uh is more likely according to the restriction that the number aspe aspect is limited. And and so it the it just makes sense on the uh looking at the whole set. So y I think all evaluation should take that into account and look at the whole aspect set. Like here you look at the aspect set because you look at the similarity of two distribution. An any st any kind of thing you where you un unfold the model and look at it manually, looks weird, like like in those Hoffmann papers where show you columns with words or things. Okay. Okay. Okay. Yeah. Yeah. Yeah. Yeah. But also it depends on that could be a good effect. Because if you if you can be if you can if you are uh highly reliable about one page, but all the page linking to it are actually about very different topics, so you could say maybe I was wrong from deciding only due to the text that this page was about this aspect distribution, because because all the other pages are voting in different direction. While there's a balance Hmm. Yeah. Pedro's always speaks about cars. But I think for any kind of way you have to select those alpha you you have to look at the effect on on the re on the t evaluation task. And then maybe you can find some patterns that are then then once you have the once you have the task, you could look at at which pages are completely missed by one model, which others are are successful with one model, and and you can maybe infer some some rules afterwards. But if you have no task, you can uh blindly select s good alphas. Like like if if you look at uh if you look at your task and you see that, I dunno uh, the the long pages for example have good P_L_S_A_ model without fusing links, so then you can infer some heuristics like okay, the alpha should be proportional to the lengths or anything like that, or to the number of image in the pages, or anything. But you should have some some task where you can individually look at the performance on every on every page. And and and and at as a second step you could infer some kind of rule about how to select alphas with respect to the task. Maybe it's v it's very simple rules, but select them blindly s might be odd. Yeah. Yeah. Yeah. Yeah. Yeah. But but but still li like the the the work I did on on hyperlinks, like I used uh Wikipedia, i I think it's g because it has fairly high level of inside links between Yeah. Well you can categorise them because they are th it's there's a structure on the on the web page. So you you can select only the links which are within the article or within the categories or it's it's it's well-structured, it's it's uh easy to manipulate and Yeah. You can even isolate the directory from the categories of Wikipedia. Well yeah, actually I I I divided in three equal size parts which have one hundred fifty documents. There are a thousand documents. And well, but Well, what I did with the link well, but this is not th exactly the same setup you would have. So in this setup, the whole link structure could be known, and the only thing you wou don't you w don't know would be the assignation to uh to a category. In my setup it was different, because I I sh I couldn't use the links between different sets. But this is not I don't want to mess you up with that, it's not the same kind of experiment. But you could just use the whole corpus like this, and and for some of them you assume you don't know the the category assignment. And then you can run this set of experiments over Wikipedia. Yeah. But this is the yeah, this is the hypothesis between this uh in this P_L_S_A_ with link stuff. Oh yeah, I thi Yeah. But you can you can think also at at other uh task like that. Like for example you can take proceedings and assume the categories are the keywords of the of the d of the documents, and then if you have the archive of a journal, you can you can use the l the links between uh the citation links. So if you want to have a bigger stuff so Wikipedia would be kind of very small with lots of Engli in links, and then if you take a journal you will have less links and and more documents. And then you would be likely to say that P_L_S_A_ might be better over that, because you don't have enough links. So you you you can find examples which are not the web I think it's good to work in closed set. Wikipedia is kind of closed set, ma even if there's lots of outside links, but there's lots of inside links. You have to think at at at at at at at database where there's lot of in links. Wikipedia might be one, citation, for example if if you take the NIPS arch archive, which is available. NIPS author always other NIPS author, which are often themselves, etcetera. Well, it's reported, I I won't say. So but but but but yeah, I think th th this kind of setup maybe web pages it's hard to find, sets of web pages which have good links stuff if you start from one page or something. Okay. But it's discrete uh no, the Okay. Yeah. Yeah. 'Kay, yeah, thanks. Okay, yeah. Well there are lots of good ideas that are recorded and someone will Yeah. Because the data like will be released very soon.